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LONG-TERM  GOALS 


My  long  term  goal  is  to  improve  the  effectiveness  of  METOC  information  flow  to  the  tactical 
users. This  infromation  flow  includes  both  the  data  flow  from  the  data  sources  to  the  METOC 
forecasters  and  the  environmental  impact  information  flow  to  the  warfighters. 

OBJECTIVES 

Improve  the  workflow  of  METOC  human-system  interaction  for  generating  useful  MEOC  products  to 
warfighters.  Develop  a  prototype  information  interaction  system  to  create  a  more  efficient  system  for 
METOC  personnel  to  present  concise  tactical  weather  information  to  the  warfighters.  Specifically, 
prototype  the  information  interaction  system  for  the  STRIKE  warfare  users.  Improve  visualization 
tools  for  presenting  tactical  information  and  provide  guidance  to  information  delivery. 

APPROACH 

A  team  of  cognitive  scientists,  METOC  researchers,  and  system  engineers  take  the  iterative  observe- 
and-build-and-test  approach  to  the  problem.  We  apply  this  approach  to  a  series  of  experiments  where 
teams  of  forecasters,  working  in  a  simulated  carrier  METOC  office  (CV-METOC),  attempt  to  provide 
a  pre-strike  mission  briefing  package.  We  will  observe  the  two-person  forecast  team  in  scenario  based 
simulation  producing  the  weather  forecast  for  the  mock  strike  mission.  We  use  cognitive  research 
tools  to  analyze  the  observations  during  the  experiment.  We  construct  the  information  flow  model 
from  the  analyses  of  the  observations,  and  we  test  our  hypotheses  in  the  following  experiments.  We 
also  develop  visualization  tools  for  assisting  the  information  flow  and  generating  tactical  products. 

WORK  COMPLETED 

This  research  project  is  a  collaborative  effort  among  several  research  groups  specialized  in  cognitive 
research,  meteorological  and  oceanographic  information  systems,  and  Strike  mission  planning 
research.  During  this  past  year,  we  have  analyzed  the  observations  collected  during  the  1999 
Experiment  conducted  in  July  1999.  The  analyzed  result  was  used  as  the  base  to  formulate  our 
hypotheses  for  the  2000  Experiment,  which  was  conducted  in  July  2000. 

1999  Experiment  Analysis 

We  observed  that  all  forecasters  made  their  forecast  decisions  from  their  mental  understanding  of  the 
weather  system.  Their  forecasts  are  quantitative,  but  their  mental  model  is  qualitative  in  nature.  Erom 
the  detailed  analysis,  we  suggested  that  METOC  forecasters  formulate  their  mental  models  and  make 
their  forecast  in  four  distinct  steps: 

1)  Initializing  their  mental  model  with  large-scale  weather  picture  particularly  significant  weather 
events, 

2)  Constructing  Qualitative  Mental  Model  (QMM)  with  weather  and  trends  in  their  region  of 
responsibility  (most  difficult  task). 


3)  Verifying  and  adjusting  QMM  against  other  information  sources  (satellite  image,  another 

forecaster,  text  forecast,  etc.),  and  finally 

4)  Constructing  the  weather  forecast  itself. 

The  details  of  this  work/information  flow  has  been  documented  and  published. 

2000  Experiment 

The  2000  Experiment  had  two  separate  parts.  They  were  conducted  simultaneously  from  two  locations 
using  the  same  strike  scenarios.  The  Experiment  was  conducted  at  NPMOC  San  Diego  and  aboard 
USS  Carl  Vinson.  At  NPMOC  San  Diego,  based  on  our  1999  Experiment  conclusions,  we  tested  the 
hypotheses  that  1)  forecasters  extract  a  quantitative  forecast  from  their  QMM,  and  2)  usage  of  certain 
tools  (mesoscale  weather  prediction  model)  can  improve  the  work/information  flow.  We  successfully 
observed  three  teams  of  forecasters  and  technicians  building  first  a  mission  planning  brief  and  later  a 
strike  briefing  for  simulated  strikes  on  Port  Angeles,  Washington  and  the  bridge  to  Whidbey  Island  for 
a  total  of  six  sessions.  In  addition,  we  observed  an  “expert”  from  the  center  at  NPMOC  prepare  a  strike 
brief.  Two  of  the  teams  were  from  the  U.S.S.  Constellation,  which  was  in  port  in  San  Diego  after  its 
recent  deployment.  One  team  was  from  the  METOC  center  at  NPMOC.  The  NPMOC  Science  Officer 
played  the  role  of  INTEE  by  providing  pre-scenario  briefings  and  on-call  support  during  the  exercise. 
NPMOC  San  Diego  also  provided  regional  center  responses  to  the  forecast  teams’  weather  questions 
and  queries.  The  observation  methodology  included  video  cameras  recording  of  computer  screens  used 
by  the  forecaster  and  the  technician,  voice  input  of  their  current  tasks  (provided  by  forecasters  and 
technicians  themselves),  notes  taken  by  the  observers  regarding  the  task  functions,  real-time  task 
recording  of  their  computer  interface  activities,  and  eye  -tracking  recording  in  an  attempt  to  capture 
the  forecaster's  cognitive  processing  from  the  computer  screens.  In  preparation  of  the  experiment,  we 
created  a  strike  specific  visualization  aid  to  test  if  it  could  help  the  forecasters  gathering  strike  specific 
information.  In  addition,  we  created  a  draft  of  workflow  checklist  to  test  if  forecasters  should  follow 
the  checklist  item  during  the  course  of  forecasting.  We  observed  that  the  experience  dictates  the  tools 
the  forecaster  uses.  The  more  experience  forecasters  used  the  mesoscale  model  more  skillfully,  and 
validated  their  mental  model  from  various  information  sources. 

We  also  conducted  the  experiment  aboard  aircraft  carrier  U.S.S.  Carl  Vinson  (CVN  70).  As  the 
experiment  carried  out  at  NPMOC  San  Diego,  we  also  successfully  observed  six  teams  of  forecasters 
and  technicians  building  strike  briefings  for  the  same  scenarios.  The  teams  were  formed  from  the 
meteorology  and  oceanography  division  (i.e.,  the  "OA"  division)  aboard  the  VINSON.  There  were 
three  different  forecasters  and  eight  different  technicians  employed  in  different  teaming  combinations. 
The  observations  were  taken  in  their  normal  operating  environment  in  the  OA  division  spaces  on  the 
third  deck  in  the  island  of  the  VINSON.  After  some  experimentation  we  believe  that  we  developed  a 
successful  methodology  and  that  we  know  how  to  deploy  with  the  VINSON  at  sea.  It  is  a  difficult  task 
because  the  space  is  very  noisy  with  more  than  seven  possible  computer  screens  to  monitor. 

The  VINSON  environment  is  very  different  than  the  NPMOC  San  Diego  study.  There  are  more 
technicians  involved  that  require  more  coordination.  It  is  also  clear  that  forecasting  is  a  continuous 
process  and  the  specific  air  strike  components  are  better  integrated  when  the  forecasters  have  this 
continuous  process  already  in  place. 


While  there  are  a  number  of  analyses  that  are  on  going,  we  observed  that  the  use  of  the  recently 
deployed  mesoscale  atmospheric  model  (TAMS/RT)  improved  when  high-level  guidelines  are 
provided  that  specifically  call  for  the  use  of  a  mesoscale  atmospheric  model.  However,  while  local 
observations  are  used  to  validate  the  mesoscale  model,  forecasters  still  need  to  consider  the 
relationship  of  the  large-scale  initialization  to  the  mesoscale  models  in  order  to  properly  forecast  using 
the  mesoscale  model. 

RESULTS 

This  important  collaboration  brings  together  researchers  from  many  disciplines  to  study  the  critical 
problem  of  information  flow  from  METCO  to  Strike  planners  by  focusing  on  the  nature  of  workflow. 

In  the  2000  Experiment,  we  have  progressed  our  research  through  carefully  designed  steps.  We  have 
advanced  our  theory  on  the  METOC  workflow,  and  we  have  fitted  our  observation  methodology  for 
use  on  board  an  aircraft  carrier. 

The  objectives  of  2000  Experiment  at  NPMOC  San  Diego  were  to  validate  workflow  concept  through 
QMM  and  to  test  if  mesoscale  weather  prediction  models  are  useful  for  forecasters  to  get  tactical 
weather  information.  We  believe  that  the  workflow  model  will  be  the  key  for  building  the  future 
METOC  information  infrastructure.  During  the  experiment,  our  observations  have  suggested  several 
significant  improvements  in  the  METOC  information  flow.  We  believe  that  1)  data/information 
staging  for  the  forecasters  is  a  key  area  for  improvement  of  the  workflow,  2)  the  modified  checklist 
approach  can  help  forecasters  to  make  the  proper  “Verify  and  Adjust”  step  in  the  QMM,  and  3)  our 
scenario  based  observation  methodology  is  a  very  good  method  for  discovering  workflow  problems  in 
other  disciplines. 

The  primary  objective  of  the  2000  Experiment  on  board  U.S.S.  Carl  Vinson  (CVN  70)  was  to  develop 
and  test  a  methodology  for  observing  Navy  weather  forecasters  produce  information  to  support  a 
simulated  air  strike  mission.  The  VINSON  is  currently  at  the  dock  in  the  Bremerton  Naval  shipyard 
during  re-fit  and  offered  a  convenient  opportunity.  Previous  Human-Systems  Information  Interaction 
studies  have  occurred  in  a  very  controlled  environment  at  a  regional  METOC  center.  The  VINSON 
study  begins  to  address  the  issues  of  the  real-world  forecaster  who  is  constrained  in  space,  time,  and 
information  sources.  Our  intent  is  to  conduct  observations  during  the  VINSON's  work-up  towards 
deployment  and  study  the  forecaster  in  near  combat  conditions.  This  study  is  the  first  of  several  to 
follow  over  the  next  nine  months. 

IMPACT/APPLICATION 

The  single  most  important  accomplishment  was  the  validation  of  the  workflow  process  through  the 
concept  of  Qualitative  Mental  Model  (QMM).  Though  the  concept  of  the  QMM  for  the  1999 
Experiment  is  still  valid,  the  details  of  this  concept  will  need  to  be  modified.  During  2000  Experiment, 
we  observed  that  forecasters  carried  this  mental  model  with  them.  We  observed  that  all  forecasters 
could  recall  details  of  their  weather  forecasts  including  quantitative  values.  But,  they  could  not  recall 
the  tactical  information  very  well.  When  the  default  forecast  values  were  provided,  forecasters  did  not 
pay  any  attention  to  those  default  values. 


Using  the  checklist  to  capture  the  QMM  can  help,  but  it  must  be  modified  for  the  following  reasons. 

1)  On  board  ship,  forecasters  make  about  20  briefs  a  day.  This  means  that  they  are  constantly 
under  pressure  to  produce  a  “forecast”  for  a  specific  need.  The  “initialize”  and  “build  a 
QMM”  steps  are  not  clearly  separated.  The  “Verify  and  Adjust”  step  is  repeated  all  day 
long. 

2)  Forecasters  and  technicians  do  work  as  a  team,  and  they  constantly  interrupt  each  other  to 
exchange  information.  Each  interruption  will  need  some  re-focus  time  for  the  forecasters  to 
get  back  to  the  task  at  hand. 

3)  In  addition  to  the  interruptions,  the  web  pages  do  not  appear  instantly  when  needed. 
Forecasters  do  not  have  the  time  or  the  patience  to  wait  for  the  specific  page  to  appear  on 
the  screen.  They  often  seek  the  required  information  from  other  web  sites,  or  they  initiate 
the  acquisition  of  other  required  information.  The  net  result  is  that  they  have  wasted  some 
of  their  valuable  time. 

The  above  three  observations  suggest  that  we  cannot  make  a  rigid  checklist  describing  the  QMM  for 
the  forecasters  to  step  through.  Instead,  we  suggest  that  the  checklist  be  re-directed  or  re-formatted  into 
a  series  of  reminders.  So,  the  checklist  will  be  flexible  enough  to  fit  the  METOC  workflow.  To  cut 
down  the  information-seek  time,  we  suggest  that  the  data  or  web  site  content  to  be  pre-staged  for  the 
forecasters.  While  there  are  a  number  of  analyses  still  on  going,  the  workflow  information  interaction 
model  begins  to  take  shape. 

TRANSITIONS 

Analyses  of  the  workflow  that  is  used  to  generate  forecasts  for  strike  warfare,  and  elements  of  the 
interaction  model  will  be  used  in  the  6.3  Environmental  Visualization  (EVIS). 

The  analysis  of  the  regional  center  web  sites  has  been  shared  with  the  METOC  community. 

RELATED  PROJECTS 

•  Personal  Simulation  System,  NRE,  $200K/yr,  10/1998  -  9/2001  (Balias) 

•  Neural  Network  Declutter  Tool:  Incorporating  Cognitive  Models  into  Decision 
Support  Systems,  ONR,  $457,502,  4/2000  -  10/2002  (Marshall) 

•  Measuring  and  Understanding  Cognitive  Workload,  MURI,  AFOSR  (subcontract 
with  George  Mason  University),  $210K/yr,  5/1997  -  4/2002  (Marshall) 

•  Shipboard  Tactical  Atmospheric  Forecast  Capability,  SPAWAR,  $450K/yr,  10/1/97- 
9/30/04  (Cook) 
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